The following table summarizes functionalities of replay buffers.

cpprb OpenAI/Baselines Ray/RLlib Chainer/ChainerRL DeepMind/Reverb
Flexible Environment Yes No No Yes Yes
Nstep Yes No Yes Yes No
Parellel Exploration Yes (Support Ape-X on single machine) Yes (Avarages gradients of MPI processes) Yes (Concatenates sample batches from distributed buffers) No Yes
Save/Load Only transitions (document) No (Maybe can pickle) No (Maybe can pickle. Trained policies can save/load.) Yes Yes
Deep Learning Framework Anything TensorFlow 1.14 (only this version) Anything (Helper functions for TensorFlow and PyTorch) Chainer (maintenance only) TensorFlow 2.3